Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

adds entity to PLE calls in cloudwatch logs plugin when used in EKS with kubernetes filter #2

Merged
merged 83 commits into from
Oct 22, 2024

Conversation

nathalapooja
Copy link

  • Modified Kubernetes filter to extract additional resource attributes
  • Modified cloudwatch logs plugin to add entity in PLE calls

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

  • Example configuration file for the change
application-log.conf: |
          [INPUT]
            Name                tail
            Tag                 application.*
            Exclude_Path        /var/log/containers/cloudwatch-agent*, /var/log/containers/fluent-bit*, /var/log/containers/aws-node*, /var/log/containers/kube-proxy*
            Path                /var/log/containers/*.log
            multiline.parser    docker, cri
            DB                  /var/fluent-bit/state/flb_container.db
            Mem_Buf_Limit       50MB
            Skip_Long_Lines     On
            Refresh_Interval    10
            Rotate_Wait         30
            storage.type        filesystem
            Read_from_Head      ${READ_FROM_HEAD}
          
          [INPUT]
            Name                tail
            Tag                 application.*
            Path                /var/log/containers/fluent-bit*
            multiline.parser    docker, cri
            DB                  /var/fluent-bit/state/flb_log.db
            Mem_Buf_Limit       5MB
            Skip_Long_Lines     On
            Refresh_Interval    10
            Read_from_Head      ${READ_FROM_HEAD}
          
          [INPUT]
            Name                tail
            Tag                 application.*
            Path                /var/log/containers/cloudwatch-agent*
            multiline.parser    docker, cri
            DB                  /var/fluent-bit/state/flb_cwagent.db
            Mem_Buf_Limit       5MB
            Skip_Long_Lines     On
            Refresh_Interval    10
            Read_from_Head      ${READ_FROM_HEAD}
          
          [FILTER]
            Name                aws
            Match               application.*
          
          [FILTER]
            Name                kubernetes
            Match               application.*
            Kube_URL            https://kubernetes.default.svc:443
            Kube_Tag_Prefix     application.var.log.containers.
            Merge_Log           On
            Merge_Log_Key       log_processed
            K8S-Logging.Parser  On
            K8S-Logging.Exclude Off
            Labels              Off
            Annotations         Off
            Use_Kubelet         On
            Kubelet_Port        10250
            Buffer_Size         0
            Use_Pod_Association On
          
          [OUTPUT]
            Name                cloudwatch_logs
            Match               application.*
            region              ${AWS_REGION}
            log_group_name      /aws/containerinsights/${CLUSTER_NAME}/application
            log_stream_prefix   ${HOST_NAME}-
            auto_create_group   true
            extra_user_agent    container-insights
            add_entity          true
  • Debug log output from testing the change
│ [2024/10/15 19:40:54] [ info] [filter:kubernetes:kubernetes.1] fetch pod to service map  │
│ [2024/10/15 19:40:54] [ info] [filter:kubernetes:kubernetes.1] [kubernetes] upstream object for pod association is NULL. Making a new one now  │
│ [2024/10/15 19:40:54] [ info] [filter:kubernetes:kubernetes.1] Request (uri = /kubernetes/pod-to-service-env-map) http_do=0, HTTP Status: 200  │
│ [2024/10/15 19:40:54] [ info] [filter:kubernetes:kubernetes.1] HTTP response payload : {"billing-service-python-6cf6c77d6d-pxhjn":{"ServiceName":"billing-service-python","Environment":"eks:test-agent/default","ServiceNameSource":"Unknown"},"insurance-service-python-6b86f768f-k │
│ wlql":{"ServiceName":"insurance-service-python","Environment":"eks:test-agent/default","ServiceNameSource":"Unknown"},**"pet-clinic-frontend-java-6d9b8667d7-zc7cl":{"ServiceName":"pet-clinic-frontend-java","Environment":"eks:test-agent/default","ServiceNameSource":"K8sWorkload"**} │
│ ,"visits-service-java-655896dbd8-nmt77":{"ServiceName":"visits-service-java","Environment":"eks:test-agent/default","ServiceNameSource":"K8sWorkload"}}  │
│  │
│ [2024/10/15 19:40:54] [ info] [filter:kubernetes:kubernetes.1] Updating pod to service map after 60 seconds
Screenshot 2024-10-15 at 3 47 37 PM
  • Attached Valgrind output that shows no leaks or memory corruption was found
==8923== 
==8923== HEAP SUMMARY:
==8923==     in use at exit: 1,584 bytes in 1 blocks
==8923==   total heap usage: 2,292 allocs, 2,291 frees, 1,009,774 bytes allocated
==8923== 
==8923== LEAK SUMMARY:
==8923==    definitely lost: 0 bytes in 0 blocks
==8923==    indirectly lost: 0 bytes in 0 blocks
==8923==      possibly lost: 0 bytes in 0 blocks
==8923==    still reachable: 1,584 bytes in 1 blocks
==8923==         suppressed: 0 bytes in 0 blocks
==8923== Reachable blocks (those to which a pointer was found) are not shown.
==8923== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==8923== 
==8923== For lists of detected and suppressed errors, rerun with: -s
==8923== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
SUCCESS: All unit tests have passed.
==3470== 
==3470== HEAP SUMMARY:
==3470==     in use at exit: 0 bytes in 0 blocks
==3470==   total heap usage: 2 allocs, 2 frees, 2,608 bytes allocated
==3470== 
==3470== All heap blocks were freed -- no leaks are possible
==3470== 
==3470== For lists of detected and suppressed errors, rerun with: -s
==3470== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

  • Run local packaging test showing all targets (including any new ones) build.
  • Set ok-package-test label to test for all targets (requires maintainer to do).
    N/A
    Documentation
  • Documentation required for this feature

Backporting

  • Backport to latest stable release.
    N/A

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

matthewfala and others added 30 commits August 20, 2024 04:29
Resolves this issue: fluent#6821

Signed-off-by: Wesley Pettit <[email protected]>
prepare_destroy_conn calls mk_list_del on the event in the
priority bucket queue, so for safety, we inject it after
this covers us against the case where the event was triggered
and waiting, but we also reached the timeout.

Signed-off-by: Wesley Pettit <[email protected]>
… everywhere with MK_EVENT_ZERO

Signed-off-by: Wesley Pettit <[email protected]>
this message is very useful in debugging Fluent Bit log loss
or frozen. But it needs context on which input the append
happened for.

Signed-off-by: Wesley Pettit <[email protected]>
These messages are useful for debugging except with only
the inode, it you have to do work to match this with
specific files.

Signed-off-by: Wesley Pettit <[email protected]>
Copy link

@swapneils swapneils left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some blocking questions and code-changes, please take a look.

Non-blocking points are prefixed as "Nit"

plugins/filter_aws/aws.c Outdated Show resolved Hide resolved
plugins/filter_kubernetes/kube_meta.c Outdated Show resolved Hide resolved
plugins/filter_kubernetes/kube_meta.c Show resolved Hide resolved
plugins/filter_kubernetes/kube_meta.c Show resolved Hide resolved
plugins/filter_kubernetes/kube_meta.c Outdated Show resolved Hide resolved
plugins/out_cloudwatch_logs/cloudwatch_api.c Outdated Show resolved Hide resolved
plugins/out_cloudwatch_logs/cloudwatch_api.c Show resolved Hide resolved
plugins/out_cloudwatch_logs/cloudwatch_api.c Show resolved Hide resolved
plugins/out_cloudwatch_logs/cloudwatch_logs.c Show resolved Hide resolved
tests/runtime/filter_kubernetes.c Show resolved Hide resolved
@swapneils swapneils merged commit 027f63c into 1.9.10 Oct 22, 2024
9 of 15 checks passed
swapneils pushed a commit that referenced this pull request Oct 22, 2024
…ch Agent

Adds Use_Pod_association option. When running in a kubernetes container, setting this option to On vends an `entity` object with pod metadata from CWA to a cloudwatch_logs output with the new `add_entity` option set to true.

(PR #2 in amazon-contributing/upstream-to-fluent-bit)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.